Methods, systems, and media for presenting supplemental content relating to media content based on state information that indicates a subsequent visit to the content interface

ABSTRACT

In accordance with some implementations of the disclosed subject matter, mechanisms for presenting recommended media content items are provided. In some implementations, a method for providing media guidance is provided, the method comprising: receiving a request for supplemental content relating to a program being presented by a media device; causing a content interface to be presented on a mobile device; identifying the program that is being presented on a media device; determining keywords associated with the program, wherein each of the keywords is associated with a particular portion of the program; obtaining and storing supplemental content items corresponding to the program based on the keywords; causing the content interface to present a first supplemental content item; receiving state information associated with the mobile device that indicates a subsequent visit to the content interface; and causing the content interface to present a second supplemental content item based on the state information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/345,342, filed Nov. 7, 2016, which is a continuation of U.S. patentapplication Ser. No. 14/190,999, filed Feb. 26, 2014, which claims thebenefit of U.S. Provisional Patent Application No. 61/922,212, filedDec. 31, 2013, each of which is hereby incorporated by reference hereinin its entirety.

TECHNICAL FIELD

The disclosed subject matter relates to methods, systems, and media forpresenting supplemental content relating to media content.

BACKGROUND

While watching a program, a viewer is often interested in informationrelating to the program. For example, the viewer may want to search foradditional information about a topic that is discussed in the program.

Search engines allow the viewer to search on the Internet for resources,which include webpages, images, video content, and audio content. Atypical search engine provides a web page for entering search terms,where the search terms are inputted by the viewer. Search resultsconducted for particular search terms are provided to the viewer.Moreover, with the advent of mobile devices, the use of mobile searchengines has increased. For example, using a mobile device, the viewermay perform a search and browse through Internet content while watchingthe program.

Text-based search engines, however, may be somewhat limited for thisuser. In an example where the user wants to find an article similar tothat which is being discussed during a program, the user may attempt toobtain this information by entering one or more keywords into a searchengine. The user then scans through search results to find a webpagecontaining information relating to the topic of the discussion. This canbe a time consuming procedure for the user. Moreover, this can cause theuser to miss a substantial portion of the program or cause the user topause the program to obtain such information via this procedure.

Accordingly, mechanisms for presenting supplemental information relatingto media content are provided.

SUMMARY

In accordance with some implementations of the disclosed subject matter,mechanisms for presenting supplemental content relating to media contentare provided.

In accordance with some implementations of the disclosed subject matter,a method for providing media guidance is provided, the methodcomprising: receiving, using a hardware processor, a request forsupplemental content relating to a program being presented by a mediadevice, wherein the request includes media information generated by themedia device presenting the program; causing a content interface to bepresented on a mobile device in response to receiving the first request;identifying the program that is being presented on a media device basedon the media information; determining a plurality of keywords associatedwith the program, wherein each of the keywords is associated with aparticular portion of the program; obtaining and storing a plurality ofsupplemental content items corresponding to the program based on theplurality of keywords; causing the content interface to present a firstsupplemental content item from the plurality of supplemental contentitems, wherein the first supplemental content item is selected based onthe keywords associated with the particular portion of the program;receiving state information associated with the mobile device thatindicates a subsequent visit to the content interface; and causing thecontent interface to present a second supplemental content item from theplurality of supplemental content items that relates to the program thatwas identified as being presented on the media device based on the stateinformation, wherein the first supplemental content item is selectedbased on the keywords associated with the particular portion of theprogram.

In accordance with some implementations of the disclosed subject matter,a system for providing media guidance is provided. The system comprisinga hardware processor that is configured to: receive a request forsupplemental content relating to a program being presented by a mediadevice, wherein the request includes media information generated by themedia device presenting the program; cause a content interface to bepresented on a mobile device in response to receiving the first request;identify the program that is being presented on a media device based onthe media information; determine a plurality of keywords associated withthe program, wherein each of the keywords is associated with aparticular portion of the program; obtain and store a plurality ofsupplemental content items corresponding to the program based on theplurality of keywords; cause the content interface to present a firstsupplemental content item from the plurality of supplemental contentitems, wherein the first supplemental content item is selected based onthe keywords associated with the particular portion of the program;receive state information associated with the mobile device thatindicates a subsequent visit to the content interface; and cause thecontent interface to present a second supplemental content item from theplurality of supplemental content items that relates to the program thatwas identified as being presented on the media device based on the stateinformation, wherein the first supplemental content item is selectedbased on the keywords associated with the particular portion of theprogram.

In accordance with some implementations of the disclosed subject matter,a non-transitory computer-readable medium containing computer-executableinstructions that, when executed by a processor, cause the processor toperform a method for providing media guidance, is provided. The methodcomprising: receiving a request for supplemental content relating to aprogram being presented by a media device, wherein the request includesmedia information generated by the media device presenting the program;causing a content interface to be presented on a mobile device inresponse to receiving the first request; identifying the program that isbeing presented on a media device based on the media information;determining a plurality of keywords associated with the program, whereineach of the keywords is associated with a particular portion of theprogram; obtaining and storing a plurality of supplemental content itemscorresponding to the program based on the plurality of keywords; causingthe content interface to present a first supplemental content item fromthe plurality of supplemental content items, wherein the firstsupplemental content item is selected based on the keywords associatedwith the particular portion of the program; receiving state informationassociated with the mobile device that indicates a subsequent visit tothe content interface; and causing the content interface to present asecond supplemental content item from the plurality of supplementalcontent items that relates to the program that was identified as beingpresented on the media device based on the state information, whereinthe first supplemental content item is selected based on the keywordsassociated with the particular portion of the program.

In accordance with some implementations of the disclosed subject matter,a system for providing media guidance is provided. The systemcomprising: means for receiving a request for supplemental contentrelating to a program being presented by a media device, wherein therequest includes media information generated by the media devicepresenting the program; means for causing a content interface to bepresented on a mobile device in response to receiving the first request;means for identifying the program that is being presented on a mediadevice based on the media information; means for determining a pluralityof keywords associated with the program, wherein each of the keywords isassociated with a particular portion of the program; means for obtainingand means for storing a plurality of supplemental content itemscorresponding to the program based on the plurality of keywords; meansfor causing the content interface to present a first supplementalcontent item from the plurality of supplemental content items, whereinthe first supplemental content item is selected based on the keywordsassociated with the particular portion of the program; means forreceiving state information associated with the mobile device thatindicates a subsequent visit to the content interface; and means forcausing the content interface to present a second supplemental contentitem from the plurality of supplemental content items that relates tothe program that was identified as being presented on the media devicebased on the state information, wherein the first supplemental contentitem is selected based on the keywords associated with the particularportion of the program.

In some implementations, the system further comprises means for causingthe content interface to be presented in response to detecting thepresence of the media device on a local area network, wherein the mediadevice and the mobile device are connected to the local area network.

In some implementations, the system further comprises means forreceiving an audio fingerprint of audio information corresponding to theprogram being presented on the media device, wherein the program isidentified based on the audio fingerprint.

In some implementations, the system further comprises: means forextracting an audio stream from a plurality of programs; means forgenerating, for each of the plurality of programs, at least one audiofingerprint from at least a portion of the extracted audio stream thatcorresponds to one of the plurality of programs; means for storing theat least one audio fingerprint in a data source indexed by program;means for comparing the audio fingerprint with the at least one storedaudio fingerprint; and means for identifying the program that is beingpresented on the media device based on the comparison.

In some implementations, the system further comprises means for storingthe plurality of supplemental content items, the media information, andthe plurality of keywords in a data source indexed by program and timewithin the program; means for selecting the first supplemental contentitem from the data source based on a first time determined from therequest; and means for selecting the second supplemental content itemfrom the data source based on a second time determined from the stateinformation.

In some implementations, the system further comprises means forreceiving an indication that the mobile device returned from a displayinhibited state in which presentation of the content interface wasinhibited.

In some implementations, the system further comprises means forreceiving an indication that the mobile device has launched arecommendation application that provides the content interface.

In some implementations, the system further comprises means for causingthe second supplemental content item to be presented in response todetermining that the state information was received prior to the end ofthe program and in response to determining that the state informationwas received within a predetermined time from the presentation of thecontent interface.

BRIEF DESCRIPTION OF THE DRAWINGS

Various objects, features, and advantages of the disclosed subjectmatter can be more fully appreciated with reference to the followingdetailed description of the disclosed subject matter when considered inconnection with the following drawings, in which like reference numeralsidentify like elements.

FIG. 1 is a flowchart of an illustrative process for providing mediaguidance with supplemental content items using one or morerecommendation interfaces in accordance with some implementations of thedisclosed subject matter.

FIG. 2 is a flowchart of an illustrative process for selectingsupplemental content items relating to an identified media content itemin accordance with some implementations of the disclosed subject matter.

FIG. 3 is a flowchart of an illustrative process for generating arecommendation interface with updated supplemental content items inaccordance with some implementations of the disclosed subject matter.

FIG. 4 is an illustrative screen of recommended items in the form ofrecommendation cards in accordance with some implementations of thedisclosed subject matter.

FIG. 5 is an illustrative screen of recommended cards that includes aninitial supplemental content item in accordance with someimplementations of the disclosed subject matter.

FIG. 6 is an illustrative screen of a mobile device that presentsrecommended cards including an indication that the mobile device isreceiving audio data corresponding a program currently being presentedon a media device in accordance with some implementations of thedisclosed subject matter.

FIG. 7 is an illustrative screen of a mobile device that presentsrecommended cards including program-related information corresponding toan identified program and supplemental content items related to theidentified program in accordance with some implementations of thedisclosed subject matter.

FIG. 8 is an illustrative screen of a mobile device that presentsrecommended cards including program-related information corresponding toan identified program and updated supplemental content items related tothe identified program in accordance with some implementations of thedisclosed subject matter.

FIG. 9 is an illustrative screen of a mobile device that presentsrecommended cards including program-related information corresponding toan identified program, updated supplemental content items related to theidentified program, and supplemental content options in accordance withsome implementations of the disclosed subject matter.

FIG. 10 is an illustrative screen of recommended cards that includes aninitial supplemental content item in response to determining that thepreviously identified program has ended and/or a predetermined period oftime has elapsed in accordance with some implementations of thedisclosed subject matter.

FIG. 11 shows a schematic diagram of an illustrative system suitable forimplementation of the mechanisms described herein for providingsupplemental content items, such as text snippets and URLs to contentrelated to the context of the media content, in accordance with someimplementations of the disclosed subject matter.

FIG. 12 shows a detailed example of a server and one of the computingdevices of FIG. 11 that can be used in accordance with someimplementations of the disclosed subject matter.

DETAILED DESCRIPTION

In accordance with various implementations, as described in more detailbelow, mechanisms, which can include systems, methods, andcomputer-readable media, for presenting supplemental content relating tomedia content are provided.

Generally speaking, a supplemental content item from one or more contentsources can be provided in a recommendation interface. For example,multiple recommendation interfaces can be concurrently presented to theuser, where one particular recommendation interface can provide asupplemental content item related to a program currently beingpresented. In a more particular example, the supplemental content itemcan be related to the particular portion of the program currently beingpresented. Such a recommendation interface can include a supplementalcontent item, information relating to the supplemental content item(e.g., a title, a source, a thumbnail image, a preview, one or morerelevant keywords, etc.), and an option to obtain the supplementalcontent item from a particular content source. It should be noted thatother suitable recommendation interfaces can also be presented, such asinterfaces that present programs or other media content information thatthe user is likely to be interested in consuming.

In some implementations, the supplemental content item provided by aselected content source can be presented as a recommendation interfaceor a content interface. In a more particular example, the recommendationinterface can be presented as a recommendation card or any othersuitable user interface that presents information to the user. In thisexample, a user can load a client application on a mobile device, wherethe client application presents the user with an opportunity to interactwith one or more cards that include, among other things, supplementalcontent items relating to media content presented by a media playbackdevice. Additionally or alternatively to loading a client application, amedia playback device can be detected using one or more discoveryprotocols and, in response to such a detection, the user can bepresented with an opportunity to interact with one or morerecommendation cards. In a more particular example, in response todetecting that a client application has been launched or in response todetecting the presence of a media playback device, such as a televisiondevice, supplemental content items relating to media content presentedby a media playback device can be presented in one or morerecommendation cards.

In some implementations, in response to determining that a user hasre-visited the client application or re-visits a particularrecommendation card including a supplemental content item, therecommendation card can be updated to include another supplementalcontent item relating to the previously identified program. For example,in response to receiving state information from a mobile device thatindicates the user has returned to a client application orrecommendation card presented by the client application, therecommendation card can presented an updated supplemental content itemto the user that includes interesting and/or relevant details for thepreviously identified program.

In some implementations, an updated supplemental content item can bepresented on a recommendation card in response to receiving anindication that the mobile device returned from a display inhibitedstate in which presentation of the content interface was inhibited. Forexample, in response to determining that the mobile device executing theclient application has transitioned from a low power state, a sleepstate, or a power conservation state to an unlocked state, an updatedsupplemental content item can be retrieved and presented within therecommendation card, where the updated supplemental content item relatesto the previously detected program.

In some implementations, an updated supplemental content item can bepresented on a recommendation card in response to receiving anindication that the mobile device has returned and/or re-launched theclient application. For example, in response to determining that themobile device that was previously executing the client application hasreturned to presenting the client application, an updated supplementalcontent item can be retrieved and presented within the recommendationcard, where the updated supplemental content item relates to thepreviously detected program.

Although the implementations described herein generally relate topresenting a user with a recommendation interface including asupplemental content item, such as a recommendation card in anapplication executing on a mobile device, this is merely illustrative.For example, the application can be implemented on a televisionplatform. The application executing on a television device can retrievesupplemental content items in response to determining a currently viewedprogram and present the user with supplemental content items.

Turning to FIG. 1, an illustrative example 100 of a process forpresenting supplemental content items based on media content is shown inaccordance with some implementations of the disclosed subject matter. At110, a computing device can load a client application that presentssupplemental content items based on media content. For example, thecomputing device can be a mobile device (e.g., a cellular telephone, asmartphone, a tablet computing device, a wearable computing device,etc.).

It should be noted that the media content can include one or moreprograms (e.g., a news program, a talk show, a sports program, etc.)from various sources, such as programs broadcast over-the-air, programsbroadcast by a cable television provider, programs broadcast by atelephone television provider, programs broadcast by a satellitetelevision provider, on-demand programs, over-the-top programs, Internetcontent, streaming programs, recorded programs, etc.

At 115, once the client application is loaded, one or morerecommendation interfaces can be presented. For example, as shown inFIG. 4, an individual card can present the current or forecasted weatherin a location which is determined to be of interest to the user, asshown in card 410. As another example, an individual card can presentrestaurant recommendation information at a location which is determinedto be of interest to a user, as shown in card 420. As yet anotherexample, an individual card can present a suggestion of nearby events ata location which is determined to be of interest to a user, as shown incard 430. As yet a further example, an individual card can present asuggestion of sports news and/or scores which is determined to be ofinterest to a user, as shown in card 440. Presentation of cards caninclude presentation of any suitable images, icons, graphics, videos,animations, sounds, haptic feedback, or any other suitable accompanyingcontent. For example, in some implementations, presentation of a cardcan include presenting an accompanying sound (e.g., a tone, a verbaldescription of the card content, or any other sounds) or accompanyinghaptic feedback (e.g., vibration of a mobile device). Although there arefour cards in the set of cards shown in FIG. 4, any suitable number ofcards (including none) can be presented to a user.

As shown in FIG. 5, in some implementations, an initial supplementalcontent recommendation card 510 can be presented. For example,supplemental content recommendation card 510 can present an initial cardprompting the user to detect the currently presented program. In a moreparticular example, recommendation card 510 can present an initial cardthat includes the prompt “DETECT TV.”

It should be noted that, in some implementations, the initialsupplemental content recommendation card 510 can be presented inresponse to detecting the presence of a media playback device. Forexample, media device discovery can be initiated on the network to whichthe computing device (e.g., the mobile device) is connected. When thecomputing device is a mobile device, media device discovery can beinitiated by the mobile device at any suitable time and for any suitablereason or reasons. For example, the client application running on themobile device can initiate media device discovery to determine whetherthe mobile device is on the same network as a digital media device, suchas a digital media server or a digital media renderer. In anotherexample, the client application running on the mobile device caninitiate media device discovery in response to initiating the clientapplication.

In a more particular implementation, a full discovery protocol can beexecuted that causes the computing device to send a User DatagramProtocol (UDP) multicast message on the current network. In someimplementations, the UDP can include an M-Search message directed tomedia devices, such as digital media renderers and/or digital mediaservers (e.g., as defined in DLNA Guidelines, December 2011, Part 1:Architecture and Protocols” technical specification, which is herebyincorporated by reference herein in its entirety), digital mediaplayers, or any other suitable media device that outputs, processes,and/or presents media content. In some implementations, the UDPmulticast message can include an address of the device sending themessage (e.g., the network address of the computing device), and caninclude a time period during which replies are to be sent. Such a timeperiod can be any suitable time period, such as one second, two seconds,etc., and can be set based on any suitable factors. An example of simpledevice discovery protocol (SSDP) is described in “UPnP™ DeviceArchitecture 1.1” technical specification as revised Oct. 15, 2008,which is hereby incorporated by reference herein in its entirety. Insome implementations, SSDP as used herein can be a modified form of SSDPand may add elements and/or omit elements as compared to SSDP asdescribed in “UPnP™ Device Architecture 1.1”.

Referring back to FIGS. 4 and 5, although not shown in FIGS. 4 and 5, insome implementations, recommendation cards or interfaces can beinteracted with by a user. For example, a card can be removed from userinterfaces if a user is not interested or is no longer interested in theinformation presented on the card. As another example, a card can be“selected” to perform an action or present more information. As a moreparticular example, if a card which displays a supplemental content itemavailable from a particular content source is presented, the card can beselected, and in response, an action can be performed, for example,access the particular content source or an application associated withthe particular content source to obtain and present the supplementalcontent item. Cards can be interacted with in any suitable way. Forexample, in some implementations, cards can be dismissed by clicking ortapping on the card or on a “dismiss” icon (e.g., an “X” at the cornerof the card or any other suitable icon). As another particular example,in some implementations, cards can be dismissed by “swiping” or“dragging” the card off the border of user interfaces. Similarly, cardscan be selected by clicking, tapping, or any other suitable mechanism.

Referring back to FIG. 1, the client application can enter a programdetection mode at 120, which can begin by obtaining an audio sample ofthe television programming currently being presented. For example, inresponse to selecting the initial supplemental content recommendationcard 510 of FIG. 5 (e.g., the “DETECT TV” card), the client applicationcan activate an audio input device coupled to the computing device, suchas a microphone, where the audio input device captures and records theaudio sample or any other suitable audio data associated with apresented television program. In another example, the client applicationcan receive a user indication to store audio data for transmission to anaudio matching service.

In a more particular example, in response to selecting the initialsupplemental content recommendation card 510 of FIG. 5, which promptsthe user to “DETECT TV,” the client application executing on computingdevice 400 can present card 610, which indicates that computing device400 is listening to media playback device 650. More particularly, theclient application can active an audio input device coupled to computingdevice 400, where the audio input device captures and records an audiosample or any other suitable audio data associated with the currentlypresented program 650 (e.g., “The Comedy Show” program on channel 4).

In some implementations, the client application can activate an audioinput device that captures audio from its surroundings and usesappropriates filters and/or other audio enhancements to extract an audiofingerprint or filtered audio data associated with the presentedtelevision program. For example, silent portions of the audio sample canbe removed to reduce the file size of the audio sample. In anotherexample, portions of the audio sample having distortion can be removedto reduce the file size of the audio sample. In yet another example, inresponse to determining that at least a portion of the audio sample doesnot include audio data having a given quality, the client applicationcan prompt the user of the client application to record another audiosample.

In some implementations, the client application executing on thecomputing device can generate an audio fingerprint of the audio sampleusing any suitable audio fingerprinting technique at 125. The audiofingerprint can be a digital representation generated from the receivedaudio sample, where the audio fingerprint can be used to identify thesame or similar portions of audio data. Alternatively, the clientapplication can transmit the audio sample associated with the televisionprogram to a search system or an audio matching system that generates anaudio fingerprint.

At 130, the client application can transmit the obtained audio sampleand/or the audio fingerprint for analysis. For example, upon launchingthe client application and activating the audio input device, the clientapplication can automatically stream the audio sample and/or the audiofingerprint to a search system that accesses a matching service (e.g.,an audio fingerprint matching service). As described above, the clientapplication can stream the audio sample to a search system, where thesearch system generates an audio fingerprint corresponding to the audiosample and transmits the audio fingerprint to an audio fingerprintmatching service for analysis.

In some implementations, additional information can be transmitted alongwith the audio sample, such as media device information, user profileinformation, location information, user preferences, timing information,and other information that can be used to generate a search queryassociated with the audio sample of the television program.

It should be noted that in implementations described herein in which theclient application (or other mechanisms described herein) collectsinformation about a particular user, the user can be provided with anopportunity to control whether the client application collectsinformation about particular users and/or how collected user informationis used by the client application. Examples of information about a usercan include the user's interests and identifying information of the user(e.g., a user profile, user credentials, device identification, etc.).Additionally, certain information about the user can be stored locally(e.g., not shared), encrypted, and/or treated in one or more ways beforeit is stored to remove personally identifiable information. For example,the client application can store user preferences and/or user interestsfor a particular user with an anonymous user identifier (e.g., a useridentifier that is not associated with the user's name, the user'susername and/or password, the user's email address, etc.). Using thesetechniques, the user can have control over what information is collectedabout the user and/or how that information is used by the clientapplication.

Turning to FIG. 2, upon receiving the audio sample and/or otheradditional information from the client application at 210, an audiofingerprint of the received audio sample can be generated at 215. Again,the audio fingerprint can be generated by the client application or anyother suitable application and transmitted to the search system or anaudio fingerprint matching system for analysis.

At 220, the search system can compare the generated audio fingerprint tomultiple audio fingerprints. For example, the generated audiofingerprint associated with the audio sample from the client applicationcan be compared against a subset of stored audio fingerprints. Inanother example, the generated audio fingerprint associated with theaudio sample from the television program can be compared against programaudio fingerprints that are stored and indexed by media content item ina database. In a more particular example, the search system can access adatabase or other suitable storage device containing multiple audiofingerprints and filter the multiple audio fingerprints to a subset ofaudio fingerprints based on location information, user preferences, andother information received from the client application. In this example,only audio fingerprints associated with the received locationinformation can be used to compare with the generated audio fingerprint.In another example, particular audio fingerprints from the database orstorage device can be used for comparison before others, such as audiofingerprints associated with popular programs, newer audio fingerprints,audio fingerprints commonly matched, etc.

In some implementations, the search system can, for example, includecapture modules that receive and process signals from multiple sources(e.g., multiple channels). These capture modules can, for each channel,capture video screenshots at particular time intervals (e.g., every twoor three seconds) and/or generate audio fingerprints from audio data atparticular time intervals. Generally speaking, these capture modules canmonitor media content from multiple content sources and generate videoscreenshots, audio fingerprints, and/or any other suitable contentidentifier. More particularly, these capture modules can store thegenerated video screenshots, audio fingerprints, and other contentidentifiers in a storage device. For example, a capture module canmonitor channels providing broadcast television content and storegenerated audio fingerprints in a database that is indexed by programand time.

It should be noted that, in some implementations, generated videoscreenshots, audio fingerprints, and other content identifiers indexedby program and time in a database can be discarded after a predeterminedtime. For example, in response to determining that a video screenshot isassociated with a time outside of a particular time window, the searchsystem or any other suitable component can remove the video screenshotfrom the database.

At 225, it can be determined whether a match is found. If no matchingaudio fingerprint is found, the search system can return to 210 and waitto receive additional audio samples from the client application.Additionally or alternatively, the search system can transmit a messageto the client application requesting that the user restart the clientapplication, transmit another audio sample (e.g., select the “DETECT TV”prompt again), and/or input or verify particular information (e.g.,program information, such as channel number, service providerinformation, program title, etc.).

For example, the matching audio fingerprint can be associated with aprogram and the search system can use this association to determine thatthe user of client application is watching the program provided on aparticular channel. In another example, the search system can transmit anotification to the client application that the program has beenidentified. This notification can include, for example, an indicationthat the search system has successfully identified the program from thereceived audio sample. In some implementations, in response to the userof the client application determining that the identified program isincorrect, the client application can provide the user with anopportunity to indicate the correct program or transmit another audiosample.

It should be noted that, in some implementations, the client applicationcan initiate the program detection mode without capturing audioinformation or using audio fingerprints (e.g., steps 120-130 of FIG. 1and steps 210-225 of FIG. 2).

In some implementations, the search system can include identifying aprogram by detecting the program that a media playback device (which isconnected to the mobile device) is currently playing back. For example,the search system can transmit a query to the mobile device to determinewhich devices are connected to the mobile device. In response todetermining that a media playback device, such as a television device, agaming device, or a set-top box, is connected to the mobile device, thesearch system can instruct the client application to communicate withthe media playback device to determine the program that the mediaplayback device is currently presenting.

In some implementations, the search system can transmit a card 710 (ortransmit updated information for the previously presented recommendationcard) to the client application that indicates a program or other mediacontent item has been identified. For example, the client applicationexecuting on computing device 400 has identified that the program 660currently being played back on media device 650 is “The Comedy Show” asshown in card 710 of FIG. 7. In some implementations, additionalprogram-related information, such as start time, end time, currentprogress, program title, episode number, series number, channel number,and network information, can be identified and associated with theidentified program. For example, as also shown in card 710 of FIG. 7,the program title, the start time, the end time, the channel number, thenetwork call sign, and the time zone are determined and presented withinrecommendation card 710.

In some implementations, in response to identifying the program, thesearch system or any other suitable system can determine keywordsassociated with the identified program at 235 and determine whethersupplemental content items exist corresponding to the determinedkeywords at 240. For example, in response to identifying a program, thesearch system can extract keywords mentioned in the identified programwithin the past X minutes (e.g., five minutes). This can include, forexample, accessing a timeline associated with the identified program anddetermining whether one or more keywords or any other suitable entityinformation have been placed on the timeline for that period of time.

In a more particular implementation, the search system can, for example,include capture modules that receive and process signals from multiplesources (e.g., multiple television channels including televisionprograms that are broadcasted by one or more television providers).These capture modules can, for each channel, process television programsas each program is being broadcast. For example, in response toreceiving frames (e.g., frames of live television program), thesecapture modules can process the frames to determine whether at least oneof the frames includes an object of interest using one or more of: anobject recognition technique to identify objects, landmarks, places,etc. that are shown in a frame, a facial recognition technique toidentify faces of actors, actresses, and/or characters that appear in aframe, and/or optical character recognition to identify text portions,such as a telephone number, a website address, a business name, aproduct name, etc. In another example, these capture modules can extractkeywords and/or entity information that is included in subtitleinformation corresponding to the program. In yet another example, thesecapture modules can determine topics that are being discussed in aprogram (e.g., by processing audio data and/or an audio fingerprint). Ina further example, these capture modules can identify a song that isplayed in a program (e.g., by processing audio data and/or an audiofingerprint). Upon performing this analysis, the object of interest canbe associated with a content item. For example, the content item caninclude a selectable link that can be associated with an identified textportion that includes a website address or a business name. In anotherexample, the content item can be a selectable link or a text snippetcorresponding to a news article, web content, or any other suitablecontent that discusses an extracted keyword or an identified topic. Inyet another example, the content item can be a selectable link directedto a music provider for playing and/or purchasing the identified song inthe program.

In some implementations, these capture modules can store the contentitems as supplemental content items in a storage device. For example, acapture module can monitor channels providing broadcast televisioncontent and store supplemental content in a database that is indexed byprogram and time. In a more particular example, a capture module canstore a supplemental content item along with timestamped audio and imagefingerprints for every N millisecond in a database while a program isbeing broadcasted by a television provider or any other suitable contentprovider.

In a more particular implementation, the search system can determine thelast five keywords mentioned in the identified program in response toreceiving a request to provide a supplemental content item. In response,the search system can use the determined keywords to select a subset ofsupplemental content items for presentation to the user. It should benoted that the supplemental content items can be presented in anysuitable manner (e.g., all at once, one at a time until a predeterminedtime has elapsed, in order of popularity or recency, etc.). For example,the search system can rank the keywords based on popularity information(e.g., search score, mentions in social media, trending keywords, mostsearched for entities in a particular program, etc.) and obtainsupplemental content items relating to the most popular keywords. Inanother example, the search system can extract keywords mentioned withinthe program that match a keyword included in a list of particularkeywords (e.g., popular topics, popular entities, etc.) and retrievesupplemental content items relating to the extracted keywords.

Referring back to FIG. 2, in response to determining one or morekeywords associated with the identified program, the search system candetermine whether supplemental content items corresponding to thedetermined keywords are available and obtain supplemental content fromsuch a database. For example, in response to determining thatsupplemental content for a particular episode of a particular program isavailable from one or more capture modules, the search system can accessthe portion of the database corresponding to the particular episode ofthe particular program and obtain the supplemental content items. In amore particular example, the supplemental content items corresponding tothe particular episode of the program can be timestamped to relate toparticular portions of the program.

Alternatively, the search system can instruct the client application toobtain supplemental content corresponding to keywords from one or moreservers (e.g., capture servers). In response, the supplemental contentcan be locally stored in a suitable storage device associated with thecomputing device.

It should be noted that a supplemental content item can include asnippet of the content item, a thumbnail image of the content item, alink or uniform resource locator (URL) to the supplemental content(e.g., a webpage having a full article describing content mentioned in aprogram), etc.

At 245, the supplemental content item can be transmitted to the clientapplication for presenting to the user. For example, a supplementalcontent item, such as a snippet of content related to the televisionprogram that the user is currently watching, can then be transmittedback to the client application executing on the mobile device for reviewand/or selection by the user of the client application. In anotherexample, a recommendation card that includes the supplemental contentitem can be transmitted to the client application for presentation asone of the multiple recommendation cards provided by the clientapplication.

In some implementations, at 250, the supplemental content items and/orrecommendation cards that include one or more supplemental content itemscan be stored along with information associated with the identifiedprogram. For example, the supplemental content items related to theidentified program can be associated with program information (e.g.,channel, program details, keyword information, etc.). In a moreparticular example, the search system can associate and store thesupplemental content items, recommendation cards, and other mediainformation such that, in response to detecting a subsequent request forsupplemental content information relating to the previously identifiedprogram, the search system can retrieve a stored supplemental contentitem and/or any other program-related information.

Referring back to FIG. 1, the supplemental content item relating to theprogram can be presented to the user at 135. For example, the clientapplication can allow the user to scroll through different supplementalcontent items (e.g., text snippets, URLs to full articles, thumbnailimages, etc.), where the user can scroll vertically or horizontally onthe computing device. In another example, the client application canautomatically present the user with a single supplemental content itemthat corresponds to the program. In this example, the obtainedsupplemental content items can be ranked based on any suitable criterion(e.g., popularity) and presented one at a time to the user.

As shown in FIG. 7, recommendation cards 720, 730, and 740 each includea supplemental content item. For example, in response to determining thekeywords “John Smith” and “hydraulic fracking,” cards 720 and 730 thatprovide text snippets, thumbnail images, links, and/or othersupplemental content relating to “John Smith” and card 740 that providessocial media snippets relating to the topic “hydraulic fracking” can bepresented to the user of the client application.

In response to receiving and presenting the supplemental content item tothe user, the client application can allow the user to select a newsitem from multiple supplemental content items. For example, the clientapplication can detect when the user has placed a highlight region overa supplemental content item and pressed a suitable button on thecomputing device. In another example, when the computing device is amobile device with a touch screen, the client application can respond tocontact with the touch screen, such as one or more taps on the touchscreen, movement of the point of contact while maintaining continuouscontact, etc.

In accordance with some implementations, the user of the computingdevice can instruct the computing device to exit the client application.Additionally or alternatively, the computing device executing the clientapplication can enter a low power state, a sleep state, or a powerconservation state. For example, the screen of the computing device canbe turned off after a certain period of inactivity. In another example,the brightness of the screen of the computing device can be dimmed aftera certain period of inactivity. In yet another example, the computingdevice can enter a state in which one or more features of the computingdevice are inhibited or operated at less than full capacity to conservepower of a power source, such as an internal battery.

At 140, the client application can detect that the client applicationhas been revisited on the computing device. For example, the clientapplication can detect that, in response to waking up the mobile deviceby selecting a button on the mobile device and/or interacting with thescreen of the mobile device, the client application has been reactivatedand recommendation cards including a recommendation card having asupplemental content item are presented. In another example, the clientapplication can detect that a user has subsequently re-launched theclient application on the mobile device. In response to such detections,the client application can transmit a request for an updatedsupplemental content item for presentation in the recommendation card.

In some implementations, the client application can determine whetherthe program or media content item has ended at 145. In response todetermining that the previously identified program has ended, the clientapplication can return to 115, where an initial recommendation card orother suitable interface can be presented. For example, the clientapplication can present a recommendation card that prompts the user todetect another program. In a more particular example, initial card 510of FIG. 5 can prompt the user to “DETECT ANOTHER PROGRAM.”

In some implementation, the client application can determine whether apredetermined time period has elapsed for presenting supplementalcontent items at 150. For example, the client application can have adefault time period (e.g., fifteen minutes) for presenting supplementalcontent items prior to prompting the user to detect another program. Inanother example, the client application can determine a time period setby the user of the client application (e.g., in user preferencesassociated with a user account). In response to detecting that thepredetermined time period has elapsed for presenting supplementalcontent items, the client application can return to 115, where aninitial recommendation card or other suitable interface can bepresented. For example, the client application can present arecommendation card that prompts the user to detect another program. Ina more particular example, initial card 510 of FIG. 5 can prompt theuser to “DETECT ANOTHER PROGRAM.”

Alternatively, in response to determining that the program has not endedat 145 and/or that the predetermined time period has not elapsed, at150, the client application can transmit a request to the search systemfor an updated supplemental content item. It should be noted that, insome implementation, the updated supplemental content item correspondsto the previously identified program and can continue to be updateduntil the previously identified program ends or until the predeterminedperiod of time has elapsed. For example, the updated supplementalcontent item can provide contextually relevant information relating tothe identified program (e.g., based on the timestamp of the request forupdated supplemental content).

Turning to FIG. 3, an illustrative example 300 of a process forgenerating a card interface including an updated supplemental contentitem based on a previously identified program is shown in accordancewith some implementations of the disclosed subject matter. At 310, thesearch system can receive an indication of a subsequent presentation ofthe card interface. As described above, the search system can receive anindication, from the client application, that the client application hasbeen reloaded on the computing device and/or receive an indication, fromthe client application, that the client application has beenre-presented on the computing device in response to returning from a lowpower state, a sleep state, or a power conservation state. Such anindication can be included in a request for an updated supplementalcontent item.

It should be noted that, in instances where a user using clientapplication consents to the use of data such as state information (e.g.,device state information relating to a return from a low power state, asleep state, or a power conservation state and/or application stateinformation relating to a return to the client application), such stateinformation can be used to subsequently present supplemental contentitems.

In some implementations, additionally or alternatively to 145 and 150 inFIG. 1, the search system can determine whether the previouslyidentified media content continues to be presented and/or whether aparticular time period has elapsed at 320. For example, the searchsystem can determine whether the previously identified program continuesto be presented by comparing the timestamp associated with the requestfor updated supplemental content with media information associated withthe program. In another example, the search system can determine whethera predetermined time period has elapsed since the initial request forsupplemental content.

In some implementations, at 330 and 340, the search system can obtainkeywords associated with a particular portion of the identified programand obtain supplemental content items associated with the obtainedkeywords. For example, the search system can process the identifiedprogram to determine the last five keywords mentioned during thepresentation of the last five minutes of the program and obtain textsnippets, thumbnail images, and links corresponding to the determinedkeywords. Alternatively, the search system can retrieve storedsupplemental content items associated with the identified program. Forexample, as described above, the search system can associate and storethe supplemental content items, recommendation cards, and other mediainformation such that, in response to detecting a subsequent request forsupplemental content information relating to the previously identifiedprogram, the search system can retrieve a stored supplemental contentitem and/or any other program-related information. In a more particularexample, supplemental content items can be selected based on timinginformation—e.g., an initial supplemental content item based onpopularity information, a subsequent supplemental content item based oncurrent playback position of the program, etc. Timing information can,for example, be used to retrieve particular supplemental content itemsfrom a database or other suitable storage device of supplemental contentitems indexed by program and time.

At 350, the search system can generate a card interface that includes atleast one of the obtained supplemental content items. For example, thesearch system can generate and transmit a recommendation card thatincludes an updated supplemental content item for presentation to theuser (e.g., at 155 of FIG. 1). In another example, the search system cantransmit the updated supplemental content item for presentation to theuser, where the client application receives the updated supplementalcontent item and renders a recommendation card that includes the updatedsupplemental content item. In a more particular example, as shown inFIG. 8, a recommendation card 810 that includes baseball-relatedsupplemental content corresponding to previously identified program 660can replace recommendation card 740 that included fracking-relatedsupplemental content corresponding to a topic discussed in program 660.More particularly, the supplemental content included in recommendationcard 810 can include content that relates to both “John Smith,” apreviously extracted keyword (which may still be contextually relevantto the program) and “baseball,” a newly extracted keyword that iscontextually relevant to the currently presented program.

In some implementations, recommendation cards or interface can beinteracted with by a user. For example, a card can be selected toperform an action or present more information. In a more particularexample, as shown in FIG. 9, in response to identifying program 660 andpresenting program card 710 and one or more supplemental content itemsin card 810, the client application can present the user with one ormore options for interacting with the recommendation cards.

As shown, the client application can present the user with option 910 toprovide information relating to the identified program. For example, inresponse to selecting option 910, the client application can retrievefrom search system or any other suitable system program-relatedinformation. In a more particular example, the client application canpresent keyword information, entity information, all of the obtainedsupplemental content items, and/or other information relating to thepreviously detected program.

As also shown in FIG. 9, the client application can present the userwith option 920 to identify another program. For example, although theclient application is currently presenting supplemental content itemsand/or other program-related information corresponding to the previouslydetected program, the user may have changed channels on media device650. In response to selecting option 920, the client application canenter the program detection mode, which can obtain an audio sample ofthe television programming currently being presented.

As described herein, in some implementations, the supplemental contentitems associated with an identified program can continue be presented onrecommendation cards or interface until it has been detected that theprogram has ended and/or that a predetermined period of time haselapsed. For example, as shown in FIG. 10, in response to detecting thatthe previously identified program 660 has ended, the client applicationcan cause an initial supplemental content recommendation card 1010(e.g., the “DETECT TV” card) to be presented. In response to selectingthe initial supplemental content recommendation card 1010, the clientapplication can activate an audio input device coupled to the computingdevice, such as a microphone, where the audio input device captures andrecords the audio sample or any other suitable audio data associatedwith a presented television program 1020.

Turning to FIG. 11, an illustrative example 1100 of a generalizedschematic diagram of a system on which the mechanisms for presentingsupplemental content items based on media content is shown in accordancewith some implementations of the disclosed subject matter. Asillustrated, system 1100 can include one or more computing devices 1110.Computing devices 1110 can be connected by one or more communicationlinks 1108 to a communications network 1106 that can be linked via acommunications link 1104 to a server 1102. In some implementations,computing devices 1110 and server 1102 can be linked via acommunications link 1112 to one or more content sources 1114.

System 1100 can include one or more servers 1102. Server 1102 can be anysuitable server for providing access to the application, such as aprocessor, a computer, a data processing device, or any suitablecombination of such devices. For example, the application can bedistributed into multiple backend components and multiple frontendcomponents and/or interfaces. In a more particular example, backendcomponents, such as data collection and data distribution can beperformed on one or more servers 1102.

In some implementations, front-end components of the application (or aportion of the application), such as a user interface and/or channelidentification features can be performed on one or more computingdevices 1110.

In some implementations, each of the computing devices 1110 and server1102 can be any of a general purpose device such as a computer or aspecial purpose device such as a client, a server, etc. Any of thesegeneral or special purpose devices can include any suitable componentssuch as a hardware processor (which can be a microprocessor, digitalsignal processor, a controller, etc.), memory, communication interfaces,display controllers, input devices, etc. For example, computing device1110 can be implemented as a smartphone, a tablet computer, a wearablecomputer, a smart television, a set-top box, a digital media receiver, agame console, a personal computer, a laptop computer, a personal dataassistant (PDA), a home entertainment system, any other suitablecomputing device, or any suitable combination thereof.

For example, in some implementations, a program can be presented using afirst computing device 1110, such as a smart television, a set-top box,a digital media receiver, etc., and supplemental content can bepresented using a second computing device 1110, such as a mobile device,a tablet computer, a smartphone, a wearable computer, a PDA, etc.

In some implementations, content source(s) 1114 can be any suitablecontent sources, such as a cable (or telephone) television contentsource, a satellite television content source, an on-demand programcontent source, an over-the-top program content source, an Internetcontent source, a streaming program content source, any other suitablecontent source, or any suitable combination thereof.

In some implementations, communications network 1106 can be any suitablecomputer network or combination of such networks including the Internet,an intranet, a wide-area network (WAN), a local-area network (LAN), awireless network, a digital subscriber line (DSL) network, a frame relaynetwork, an asynchronous transfer mode (ATM) network, a virtual privatenetwork (VPN), etc. Communications links 1104, 1108, and 1112 can be anycommunications links suitable for communicating data among computingdevices 1110, server 1102, and/or content source(s) 714, such as networklinks, dial-up links, wireless links, hard-wired links, any othersuitable communications links, or any suitable combination of suchlinks. Computing devices 1110 can enable use of the techniques describedherein that can allow the features of the mechanisms to be used.Computing devices 1110, server 1102 and content source(s) 1114 can belocated at any suitable location.

In some implementations, server 1102 can include one or more modules1120-1130 and/or databases 1140-1142 that can be interconnected, forperforming various functions of the application and/or facilitating themechanisms for presenting supplemental content items based on mediacontent information.

In some implementations, a front-end server module 1120 can performprocesses for presenting supplemental content items as described abovein connection with, for example, FIGS. 1-3. For example, the front-endserver can act as a proxy between the client application executing oncomputing device 1110 and other server modules, such as capture module1122, audio matching module 1124, and supplemental content module 1126.

In some implementations, a capture module 1122 can receive media datarelated to a program or a channel, such as video data, audio data,electronic program guide data, metadata, subtitles or captioningcontent, etc., as described above in connection with, for example, FIGS.1 and 2. Additionally or alternatively, capture module 1122 can extractvarious media data from content provided from content sources asdescribed in connection with, for example, FIGS. 1 and 2. Such extractedmedia data can include, for example, audio fingerprints, subtitles, etc.This information can be stored, for example, in a database (not shown)for use by the search application executing on front-end server 1120 inidentifying channels, identifying program and/or other program-relatedinformation, obtaining supplemental content items, and/or various otheroperations.

In some implementations, guidance data 1140, such as an electronicprogram guide (EPG) database and a closed caption database, can beaccessed. For example, front-end server module 1120 can access guidancedata 1140 to obtain program information related to an identified channelor an identified program and use the program information to search formatching supplemental content items. It should be noted that, in someimplementations, guidance data 1140 can be provided from, for example,content source(s) 1114.

In some implementations, an audio matching module 1124 can receive audiofingerprint data for a program from capture module 1122 and compare thereceived audio fingerprint data against stored audio fingerprint data ina fingerprint database 1142. This fingerprint database can be used, forexample, to identify a channel as described above, to determine aprogram that is being presented to a user, etc.

In some implementations, a supplemental content module 1126 can findmatching supplemental content items associated with a program usingkeyword extraction module 1130 and supplemental content matching module1128. For example, keyword extraction module 1130 can extract keywordsfrom captured audio data, video data, and/or subtitle information andobtain supplemental content items from multiple content sources (e.g.,content sources 1114). In some implementations, keyword extractionmodule 1130 can then rank each of the supplemental content items byrecency, popularity (e.g., popularity based on click-through-rate), etc.In response to ranking the supplemental content items, keywordextraction module 1130 can select at least one of the supplementalcontent items for presenting to the user at a particular position in theprogram.

It should be noted that, although modules 1120-1130 and databases1140-1142 are shown as being included in server 1102, these modulesand/or databases can be provided in different servers in variouscombinations. For example, supplemental content module 1126 can beprovided on a first server and modules 1120-1124 can be provided on asecond server. As another example, each module can be provided on adifferent server. It should be noted that these are merely examples, andthe modules described can be organized in any suitable manner.

It should also be noted that modules 1120-1130 can be included as partof the application, can each be a separate application, or can be partof any other suitable application.

FIG. 12 illustrates an example 1200 of hardware that can be used toimplement server 1102 and one of computing devices 1110 depicted in FIG.11 in accordance with some implementations of the disclosed subjectmatter. Referring to FIG. 12, computing device 1110 can include ahardware processor 1212, a display 1214, an input device 1216, andmemory 1218, which can be interconnected. In some implementations,memory 1218 can include a storage device (such as a non-transitivecomputer-readable medium) for storing a computer program for controllinghardware processor 1212.

Hardware processor 1212 can use the computer program to present ondisplay 1214 content and/or an interface that allows a user to interactwith the client application and to send and receive data throughcommunications link 1108. It should also be noted that data receivedthrough communications link 1108 or any other communications links canbe received from any suitable source. In some implementations, hardwareprocessor 1212 can send and receive data through communications link1108 or any other communication links using, for example, a transmitter,receiver, transmitter/receiver, transceiver, or any other suitablecommunication device. Input device 1216 can be a computer keyboard, amouse, a trackball, a keypad, a remote control, any other suitable inputdevice, or any suitable combination thereof. Additionally oralternatively, input device 1216 can include a touch screen display 1214that can receive input (e.g. using a finger, a stylus, or the like).

Server 1102 can include a hardware processor 1222, a display 1224, aninput device 1226, and memory 1228, which can be interconnected. In someimplementations, memory 1228 can include a storage device for storingdata received through communications link 1104 or through other links,and processor 1222 can receive commands and values transmitted by one ormore users of, for example, computing device 1110. The storage devicecan further include a server program for controlling hardware processor1222.

The mechanisms described herein for providing supplemental content itemsand other content based on media content can be implemented in computingdevices 1110 and/or server 1102 as software, firmware, hardware, or anysuitable combination thereof.

In some implementations, server 1102 can be implemented as one server orcan be distributed as any suitable number of servers. For example,multiple servers 1102 can be implemented in various locations toincrease reliability, function of the application, and/or the speed atwhich the server can communicate with computing devices 1110.

In some implementations, the client application can include anapplication program interface (not shown), and/or can be resident inmemory 1218 of computing device 1110 and/or memory 1228 of server 1102.Additionally or alternatively, a graphical user interface (“GUI”) can bedistributed to computing device 1210, which can allow a user to interactwith the client application resident at, for example, server 1102.

In some implementations, the application can include client-sidesoftware, server-side software, hardware, firmware, or any suitablecombination thereof. For example, the application can encompass acomputer program that causes one or more processors to execute thecontent generation application. As another example, the application(s)can encompass a computer program written in a programming languagerecognizable by computing device 510 and/or server 502 that is executingthe application(s) (e.g., a program written in a programming language,such as, JAVA, C, Objective-C, C++, C#, JAVASCRIPT, Visual Basic, HTML,XML, ColdFusion, any other suitable approaches, or any suitablecombination thereof).

In some implementations, the application can encompass one or moreWeb-pages or Web-page portions (e.g., via any suitable encoding, such asHyperText Markup Language (“HTML”), Dynamic HyperText Markup Language(“DHTML”), Extensible Markup Language (“XML”), JAVASERVER Pages (“JSP”),Active Server Pages (“ASP”), Cold Fusion, or any other suitableapproaches).

In some implementations, any suitable computer readable media can beused for storing instructions for performing the functions and/orprocesses described herein. For example, in some implementations,computer readable media can be transitory or non-transitory. Forexample, non-transitory computer readable media can include media suchas magnetic media (such as hard disks, floppy disks, or any othersuitable magnetic media), optical media (such as compact discs, digitalvideo discs, BLU-RAY discs, or any other suitable optical media),semiconductor media (such as flash memory, electrically programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM), or any other suitable semiconductor media), anysuitable media that is not fleeting or devoid of any semblance ofpermanence during transmission, and/or any suitable tangible media. Asanother example, transitory computer readable media can include signalson networks, in wires, conductors, optical fibers, circuits, anysuitable media that is fleeting and devoid of any semblance ofpermanence during transmission, and/or any suitable intangible media.

It should be understood that the above described steps of the processesof FIGS. 1-3 can be executed or performed in any order or sequence notlimited to the order and sequence shown and described in the figures.Also, some of the above steps of the processes of FIGS. 1-3 can beexecuted or performed substantially simultaneously where appropriate orin parallel to reduce latency and processing times.

It should also be noted that, as used herein, the term mechanism canencompass hardware, software, firmware, or any suitable combinationthereof.

Accordingly, methods, systems, and media for presenting supplementalcontent relating to media content are provided.

Although the invention has been described and illustrated in theforegoing illustrative implementations, it is understood that thepresent disclosure has been made only by way of example, and thatnumerous changes in the details of the implementation of the inventioncan be made without departing from the spirit and scope of theinvention, which is limited only by the claims that follow. Features ofthe disclosed implementations can be combined and rearranged in variousways.

What is claimed is:
 1. A method for providing media guidance, the methodcomprising: causing, using a hardware processor, a plurality ofrecommendation card interfaces that each include one of a plurality ofsupplemental content items to be presented on a display associated witha mobile device, wherein each of the plurality of supplemental contentitems is stored on the mobile device as relating to a media content itembeing presented by a media device; receiving, using the hardwareprocessor, state information associated with the mobile device thatindicates a subsequent visit to the plurality of recommendation cardinterfaces; determining, using the hardware processor, that the stateinformation comprises an indication that the mobile device returned froma display inhibited state in which presentation of the plurality ofrecommendation card interfaces was previously inhibited; identifying,using the hardware processor, the media content item being presented bythe media device; and causing, using the hardware processor, theplurality of recommendation card interfaces to be updated with at leasta second supplemental content item from the plurality of supplementalcontent items stored on the mobile device, wherein the secondsupplemental content item relates to the media content item that wasidentified as being presented by the media device.
 2. The method ofclaim 1, further comprising: obtaining the plurality of supplementalcontent items corresponding to the media content item.
 3. The method ofclaim 1, further comprising causing the plurality of recommendation cardinterfaces to be presented in response to detecting the presence of themedia device on a local area network, wherein the media device and themobile device are connected to the local area network.
 4. The method ofclaim 1, further comprising receiving an audio fingerprint of audioinformation corresponding to the media content item being presented onthe media device, wherein the media content item is identified based onthe audio fingerprint.
 5. The method of claim 1, further comprising:extracting an audio stream from a plurality of media content items;generating, for each of the plurality of media content items, at leastone audio fingerprint from at least a portion of the extracted audiostream that corresponds to one of the plurality of media content items;storing the at least one audio fingerprint in a data source indexed bymedia content item; comparing the audio fingerprint with the at leastone stored audio fingerprint; and identifying the media content itemthat is being presented on the media device based on the comparison. 6.The method of claim 1, further comprising: storing the plurality ofsupplemental content items, media information, and a plurality ofkeywords in a data source indexed by media content item and time withinthe media content item; selecting the first supplemental content itemfrom the data source based on a first time determined from a request forsupplemental content relating to the media content item being presentedby the media device; and selecting the second supplemental content itemfrom the data source based on a second time determined from the stateinformation.
 7. The method of claim 1, wherein the second supplementalcontent item is caused to be presented in response to determining thatthe state information was received prior to the end of the media contentitem and in response to determining that the state information wasreceived within a predetermined time from the presentation of theplurality of recommendation card interfaces.
 8. A system for providingmedia guidance, the system comprising: a hardware processor that isconfigured to: cause a plurality of recommendation card interfaces thateach include one of a plurality of supplemental content items to bepresented on a display associated with a mobile device, wherein each ofthe plurality of supplemental content items is stored on the mobiledevice as relating to a media content item being presented by a mediadevice; receive state information associated with the mobile device thatindicates a subsequent visit to the plurality of recommendation cardinterfaces; determine that the state information comprises an indicationthat the mobile device returned from a display inhibited state in whichpresentation of the plurality of recommendation card interfaces waspreviously inhibited; identify the media content item being presented bythe media device; and cause the plurality of recommendation cardinterfaces to be updated with at least a second supplemental contentitem from the plurality of supplemental content items stored on themobile device, wherein the second supplemental content item relates tothe media content item that was identified as being presented by themedia device.
 9. The system of claim 8, wherein the hardware processoris further configured to: obtain the plurality of supplemental contentitems corresponding to the media content item.
 10. The system of claim8, wherein the hardware processor is further configured to cause theplurality of recommendation card interfaces to be presented in responseto detecting the presence of the media device on a local area network,wherein the media device and the mobile device are connected to thelocal area network.
 11. The system of claim 8, wherein the hardwareprocessor is further configured to receive an audio fingerprint of audioinformation corresponding to the media content item being presented onthe media device, wherein the media content item is identified based onthe audio fingerprint.
 12. The system of claim 8, wherein the hardwareprocessor is further configured to: extract an audio stream from aplurality of media content items; generate, for each of the plurality ofmedia content items, at least one audio fingerprint from at least aportion of the extracted audio stream that corresponds to one of theplurality of media content items; store the at least one audiofingerprint in a data source indexed by media content item; compare theaudio fingerprint with the at least one stored audio fingerprint; andidentify the media content item that is being presented on the mediadevice based on the comparison.
 13. The system of claim 8, wherein thehardware processor is further configured to: store the plurality ofsupplemental content items, media information, and a plurality ofkeywords in a data source indexed by media content item and time withinthe media content item; select the first supplemental content item fromthe data source based on a first time determined from a request forsupplemental content relating to the media content item being presentedby the media device; and select the second supplemental content itemfrom the data source based on a second time determined from the stateinformation.
 14. The system of claim 8, wherein the second supplementalcontent item is caused to be presented in response to determining thatthe state information was received prior to the end of the media contentitem and in response to determining that the state information wasreceived within a predetermined time from the presentation of theplurality of recommendation card interfaces.
 15. A non-transitorycomputer-readable medium containing computer executable instructionsthat, when executed by a processor, cause the processor to perform amethod for providing media guidance, the method comprising: causing aplurality of recommendation card interfaces that each include one of aplurality of supplemental content items to be presented on a displayassociated with a mobile device, wherein each of the plurality ofsupplemental content items is stored on the mobile device as relating toa media content item being presented by a media device; receiving stateinformation associated with the mobile device that indicates asubsequent visit to the plurality of recommendation card interfaces;determining that the state information comprises an indication that themobile device returned from a display inhibited state in whichpresentation of the plurality of recommendation card interfaces waspreviously inhibited; identifying the media content item being presentedby the media device; and causing the plurality of recommendation cardinterfaces to be updated with at least a second supplemental contentitem from the plurality of supplemental content items stored on themobile device, wherein the second supplemental content item relates tothe media content item that was identified as being presented by themedia device.
 16. The non-transitory computer-readable medium of claim15, wherein the method further comprises: obtaining the plurality ofsupplemental content items corresponding to the media content item. 17.The non-transitory computer-readable medium of claim 15, wherein themethod further comprises causing the plurality of recommendation cardinterfaces to be presented in response to detecting the presence of themedia device on a local area network, wherein the media device and themobile device are connected to the local area network.
 18. Thenon-transitory computer-readable medium of claim 15, wherein the methodfurther comprises receiving an audio fingerprint of audio informationcorresponding to the media content item being presented on the mediadevice, wherein the media content item is identified based on the audiofingerprint.
 19. The non-transitory computer-readable medium of claim15, wherein the method further comprises: extracting an audio streamfrom a plurality of media content items; generating, for each of theplurality of media content items, at least one audio fingerprint from atleast a portion of the extracted audio stream that corresponds to one ofthe plurality of media content items; storing the at least one audiofingerprint in a data source indexed by media content item; comparingthe audio fingerprint with the at least one stored audio fingerprint;and identifying the media content item that is being presented on themedia device based on the comparison.
 20. The non-transitorycomputer-readable medium of claim 15, wherein the method furthercomprises: storing the plurality of supplemental content items, mediainformation, and a plurality of keywords in a data source indexed bymedia content item and time within the media content item; selecting thefirst supplemental content item from the data source based on a firsttime determined from a request for supplemental content relating to themedia content item being presented by the media device; and selectingthe second supplemental content item from the data source based on asecond time determined from the state information.
 21. Thenon-transitory computer-readable medium of claim 15, wherein the secondsupplemental content item is caused to be presented in response todetermining that the state information was received prior to the end ofthe media content item and in response to determining that the stateinformation was received within a predetermined time from thepresentation of the plurality of recommendation card interfaces.